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ABSTRACT 

This paper describes the evaluation of a current 
awareness or selective dissemination of information (SDI) service at 
New Mexico State University. To provide expanded library services to 
faculty, a current awareness service was offered to 18 science 
departments in 1991. A profile for each researcher enrolled was 
entered into a program and was run against "Current Contents" 
diskettes weekly. Each week relevant citations were delivered to the 
faculty member who could then request the articles to be delivered. 
After 18 months of the service, a questionnaire was mailed to the 
faculty using the service. The purpose of this evaluation was to 
determine the level of satisfaction and to develop a method for 
deciding which profiles needed updating. There were 71 responses out 
of 98 profiles, for a response rate of 72%. It was concluded from the 
findings that the users were satisfied with the service; the majority 
of citations received were considered relevant; 35% of the users 
needed revision of their profiles; and few faculty members took 
advantage of the document del ivery service . The survey instrument is 
included. (JLB) 
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ABSTRACT 

The challenge of identifying relevant documents is of critical importance 
for researchers and for those in the information professions. The 
continuing proliferation of published literature increases the amount of 
time and difficulty for those who must locate, sort, and read scholarly 
publications. In order to meet this challenge a current awareness service 
was instituted at New Mexico State University in the fall of 1991. The 
value of this kind of service is based primarily on its relevancy and 
immediacy. The relevancy of retrieved documents is dependent on the 
quality of the profiles. Therefore, a means of evaluation of the service is 
necessary for the purpose of determining whether profiles are working. 
This paper discusses the importance of updating profiles and a method for 
evaluating a current awareness service as a whole. 
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CURRENT AWARENESS IN LIBRARIES: BACKGROUND 

Identification of relevant material from current literature is crucial 
to the academic researcher. The library is a partner in the identification 
and procurement process which may take the form of a current awareness 
service. Current awareness services are not new. Libraries began offering 
such services by manually scanning documents as much as 50 years ago 
(Ohta and Evans, 1970) 1 . Lavendel (1982)2 described selective 
dissemination of information (SDI) as "the quintessential library service". 
Recently, computer databases and software have become available and 
made the scanning of literature much faster than manual scanning. While 
computerized scanning is not always superior to manual scanning, the 
savings in cost and time make computerized scanning necessary. Manual 
scanning is now impractical except in some special libraries, due to the 
enormous amount of literature to be covered. Document delivery services 
are a natural link to current awareness services. Rapid, ready access to the 
literature should follow its identification. Hence, interlibrary loan and 
commercial document delivery services become part of the current 
awareness equation. Evaluation of a current awareness or SDI service is 
crucial for several reasons. Inherent in the service, indeed in the phrase 
"current awareness" is its up-to-dateness. One of its greatest values, 
therefore, is its immediacy. The ability to make changes to the service as 
appropriate, to make them frequently, and to make the researcher/users 
aware of the ease of changing profiles is highly important. For various 
reasons the profile may not be working. Subtle differences in profile 
keywords can cause either lack of retrieval on the one hand or information 
overload on the other. Additionally, research interests can and do change. 

Improvements in technology have led to improvements in service. 
Cox (1992)3 described a current awareness service that emphasized the 
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use of personal bibliographic software to manage literature citations. 
Butler (1993)4 described the use of e-mail for citation delivery and update 
requests. Evaluations of current awareness services via surveys or 
personal contacts have been described by Butler (1993) 5 , Brandli (1976)6, 
Jax and Houlson (1988) 7 , and Whitehall (1986) 8 . While these evaluations 
were able to determine levels of user satisfaction, they were not designed 
to determine whether user profiles were to be modified. This was the goal 
of the current work. 
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THE CURRENT AWARENESS SERVICE AT NEW MEXICO STATE UNIVERSITY 

New Mexico State University (NMSU) is a medium sized (15,000 
students) university with 48 masters and 24 doctoral degree programs. To 
provide expanded library services to NMSU faculty a current awareness 
service was offered to 18 science departments. Subscription to four 
Current Contents diskettes (Physical, Chemical, and Earth Sciences; 
Agriculture, Biology, and Environmental Sciences; Life Sciences; and 
Engineering and Applied Sciences) was initiated from the Institute for 
Scientific Information (ISI). 

The service was marketed in several ways. A letter describing the 
service was sent to all faculty in the sciences and when possible a librarian 
described the service at departmental faculty meetings. Enrollment forms 
similar to those used for online search requests, were distributed for the 
purposes of profiling. Meetings with individual faculty were arranged to 
develop profiles or, if this proved difficult, profiles were established via 
telephone. The service was also publicized in the NMSU Library 
Newsletter which is distributed quarterly to all faculty. Some researchers 
chose to have their profiles run against more than one diskette, eg; Life 
Sciences in addition to Agriculture, Biology, and Environmental Sciences. 
Faculty members had the option of supplying a fund number so that 
documents could be delivered for a fee of $4 each. 

The profile for each researcher was entered into the ISI program, 
stored, and run against the diskettes weekly. Emphasis was made to all 
faculty that a current awareness service is an ongoing process and that 
they might expect to make periodic revisions. Each week relevant citations 
were printed or downloaded to floppy disk and delivered via campus mail, 
or were sent by e-mail. A word processor was used to maintain a master 
list of the profiles and to keep records of weekly "hits". This provided a 
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convenient way to print a list of the names and profiles for mailing. If 
faculty wanted document delivery, they sent a marked citation list via 
campus mail or they could request articles by telephone. Requested 
articles were obtained from library shelves or from ISFs document 
delivery service (with a 4-day average turnaround time). Faculty were 
asked to contact a librarian with any concerns or questions and profiles 
were adjusted upon request. If more than 15 "hits" were obtained per 
week, the user was contacted to ascertain if the profile was too broad. 

The initial investment in time was substantial. Contacts and 
meetings for discussion of the program, discussion of research interests, 
and creating and entering the profiles, required nearly one hour per 
profile. When the librarians had the system working smoothly, a staff 
person was hired to take over the duties of running the profiles, printing, 
and mailing the results. The librarians continue to maintain all current 
awareness profiles. 

Despite distribution of enrollment forms at faculty meetings and in 
campus mail, only about 15% of the faculty were requesting enrollment. 
Follow up phone calls improved this percentage to about 30% and after two 
years about 50% of eligible faculty are enrolled in the current awareness 
service. Continued follow up phone calls and personal contacts would 
undoubtedly have kept increasing the percent of users to nearly 100%. 
However, the librarians began to reconsider whether they should continue 
to push the service on those who were not enthusiastic about it. As one 
librarian remarked ''you could talk anybody into it". Current awareness 
services are of most benefit to those active in research or writing. Among 
the reasons given for not enrolling were that some researchers preferred 
to do their own scanning in a subject index, that they didn't need a 
literature review now, that they scanned only a limited number of 
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journals, their peers keep them informed, or that they had their own 
Current Contents subscriptions. 

As more departments began to participate and word of the service 
spread we began to get inquiries from those who had heard of the service 
from others. Additionally, reference desk transactions sometimes resulted 
in referrals to the current awareness service. For the approximately 100 
science faculty currently enrolled in the program the staff assistant spends 
about 4 hours per week running the profiles and printing and mailing the 
results. The fee-based document delivery demand has been low. Of 
document delivery requests about 70% of the articles are not in our library 
and are obtained from ISI at a cost of approximately $12.50 per article. 
Initially the short form of citations was supplied but after several requests 
for author addresses all citations are now delivered in long form, which 
includes author address. The cost to the library of the ISI subscriptions 
and reprints is about $150 per month at present. 
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EVALUATION OF THE SERVICE 

After approximately 18 months a questionnaire (Appendix 1) was 
mailed to each of the faculty using the service. The purpose was to 
determine satisfaction levels and develop a method for deciding which 
profiles needed updating. We also wanted to determine how cited papers 
were being obtained. Since initial response to the questionnaire was below 
50%, a second mailing was made to those who had not responded. 
Ultimately there were 71 responses out of 98 profilees (72%). 

The first question (A, appendix I) was designed to verify that the 
responder was in fact getting the mailed citation lists and to introduce the 
term "citation/ 1 which was used in several succeeding questions. Sixty-six 
of 71 respondents answered "yes" (It was later determined that, in fact, all 
had received citations). The next two questions were used to establish a 
measure of the level of satisfaction with the quality and coverage of the 
citations. Responses to question B "What percentage of citations are 
appropriate to your topic?" ranged from 5% to 100%. The mean was 58%. 
Responses to question C "What percentage of citations you may be 
missing?" ranged from 0% to 90% with a mean of 23%. These types of 
questions are valuable in determining whether a profile is working for an 
individual or whether it needs revision. Both questions B and C, as well as 
other questions, were used to devise a list of faculty names to be contacted 
to discuss revision of profiles. For question B, the "percent of appropriate 
citations", any user who answered that they were receiving fewer than 
50% appropriate citations was listed for contacting. There were 22 such 
responses below 50%. For question C, the percent of missed citations, 
anyone who answered greater than 30% missed citations was added to the 
list. Ten of the responses indicated that more than 30% of citations may 
have been missed. In comparison, more than twice the number of users 
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were deemed unsatisfied due to receiving too many irrelevant citatons 
(question B) as missing citations (question C). 

Question D was used to determine other ways users were keeping up 
with literature in addition to the current awareness service. Responses fell 
into four categories; browsing, using indexes, personal subscriptions, and 
personal communications. The greatest number of responders (29) 
indicated that browsing in the library was a method used to stay abreast 
of current literature. Many indicated that they regularly browsed the 
current periodicals room. Some simply stated they used the library. 
Personal subscriptions was a response given by 19 of those surveyed and 
some indicated the names of those subscriptions. The use of indexes to 
keep current was indicated by 18 respondents. These included print, CD- 
ROM, and online databases, and some responders named the indexes used. 
The final category was personal communications, including meetings, e- 
mail, and other methods of contacting one's colleagues. Eleven respondents 
indicated that this was an important means of keeping current. 

The responses to question E, how does one "obtain papers from 
citations 1 ' delivered by the service, revealed that there were many ways of 
obtaining citations. Answers were ranked from "1" (most frequent) to "6" 
(least frequent). If there was no response it was assumed that that 
method was not used. In order to get an estimate of the most common 
methods of obtaining papers, all of the "l" and "2" frequency responses 
were combined for each of five categories of obtaining papers. Using this 
method it was found that papers were most frequently obtained from the 
library shelves (52 "1" or "2" responses). Papers obtained from personal 
subscriptions accounted for 30 responses. Papers were also commonly 
obtained from interlibrary loan (21 responses) or from authors (19 
responses). The current awareness document delivery service was 
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answered as frequently used ("1" or "2" response) by only 3 users. 

Question F asked whether the responder had used the document 
delivery service. Only ten answered yes. Five users responded that they 
thought the cost per article ($4) was too high. Of the 60 "no" responses, 
many indicated that they either did not know about it or did not think 
about using it. It became apparent that many of those profiled over the 
telephone had not been told of the service. Those who had filled out an 
enrollment form, on the other hand, had had the opportunity to provide a 
fund number for use in ordering articles and had theoretically, at least, 
been informed. A memorandum was sent to all users describing the 
document delivery service. Despite this memorandum, document delivery 
requests remain low. 

Question G was a general question to determine what changes in the 
service would be useful. There were 21 responses to this question. The 
majority (17) requested a change in their profile. These users were added 
to the list of those to be contacted. Other requested changes were to "make 
the service free" (it was already free, unless they were referring to 
document delivery), to "change from disk to print citation delivery" (done 
for this user), to "include abstracts" (this was done), and to "add Clinical 
Medicine Current Contents" (this was not done). 

Question H, "Have you changed your profile..." allowed us to compare 
satisfaction levels from other survey questions with previous changes in 
profiles. Eleven users answered "yes" to this question. Of these, one was 
requesting another change. There were, however, no apparent differences 
between those who had or had not had profiles changed, and their answers 
to o aestions B and C (% missing or % appropriate citations). 

The final question, question I, asked the user to rank the service 
from "1" (very useful) to "5" (not useful, please discontinue). The majority 
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of respondents (60 or 84%) answered "1" (very useful) or "2". Those who 
answered "3" or above were judged to be unsatisfied. There were eight 
H 3's'\ two "4y\ and one "5 M . The "5" was discontinued and the others 
added to the list of persons to be contacted. 

By this point there were four categories of users to be contacted. 
Those with inappropriate citations, with missing citations, who ranked the 
usefulness of the service as low, and those who specifically requested 
profile changes. These lists were merged resulting in 35 users to be 
contacted, just under 50% of those who had responded to the survey, about 
35% of the total number of users. Each individual was contacted and 
thanked for answering the survey. The specific reason for contacting the 
individual was discussed, eg. "you indicated that you were missing about 
60% of the citations". The existing profile was reviewed to determine why 
there seemed to be irrelevant or missing citations. Sometimes it was a 
simple matter of adding or dropping keywords. For more complicated 
profiles, a conference was suggested in which modification of the profile 
was done at the computer terminal and various output combinations were 
compared until the user and the librarian were satisfied with the results. 
Users were assured that the only way to make the system work to its 
potential was to continue modifying the profile until it met their needs. 
Additionally, it was stressed that peopled needs change and they should 
feel free to contact the librarian at any time to make further changes. 
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CONCLUSION 

The current awareness evaluation survey revealed that users were 
highly satisfied with the service. Respondents indicated that the majority 
of citations received were relevant to their research and that on average 
they felt they were missing about 23% of citations. The survey evaluation 
included four factors that were designed to determine if profile changes 
were needed. Using these factors, approximately 35% of the users were 
contacted for consultation and profile revision. Such revisions are a crucial 
part of any current awareness service. The current awareness service 
complements library online and CD-ROM services and has been successful 
in improving the visibility of the library. The response of faculty has been 
very positive with many compliments and several faculty members have 
approached us to enroll after hearing of the service from a colleague. It is 
expected that about 50% of the faculty will use the service. The use of e- 
mail for delivery of citations is increasing. Few faculty, however, are 
taking advantage of our fee-based delivery service. While the cost is low 
($4) and does not even require a visit to the library, it appears that most 
researchers are getting materials from our shelves, from personal 
subscriptions, using ILL, or requesting reprints directly from authors. 
Some faculty researchers have developed more insight into how our CD- 
ROM collection, our online search service, and our current awareness 
service complement each other. Several have become regular CD-ROM 
users. For a modest cost we have been able to improve our service, our 
image, and our liaison function. There has been a positive impact on 
librarians who have gained a better awareness of faculty research 
interests. 
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APPENDIX I 

Memo to: 

From: Roger Steeb, Tim McKimmie, Dept. 3475 
Subject: Current Awareness Service 
Date: April 1, 1993 

Please complete this questionnaire and return to Box 3475, 

A. Have you received weekly citation lists from the current awareness service? Yes No 

B. What percentage of citations that you receive are appropriate to your topic? % 

C. What percentage of relevant citations do you think you may be missing? % 

(ie., citations that you find out about from sources other than our current awareness service) 

D. What other sources have you used during the past 3 months to stay current? 



E. How do you generally obtain papers from citations you receive? Please rank in order of 
frequency l=most frequent, 6=Ieast frequent. 

Rank(l, 2, 3, 4, 5, 6) 

interlibrary loan 

directly from authors 

personal subscription 

library shelves 

from current awareness sendee (document delivery) 

other 



F. Do you use our document delivery service ($4 per article)? Yes No 

please comment: 



G. Are there any changes in the service that would be helpful? Please list. 



H. Have you changed your profile since starting the service? Yes No 



I. Please rank the usefulness of the current awareness service to you. (please circle) 

not useful, 

very useful please discontinue 
1 2 3 4 5 

] 4 



